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ABSTRACT 



Although adult literacy programs exist in almost every 
country throughout the developing world, comprehensive information on their 
capacity, performance, and impact is seriously lacking. This deficiency has 
direct consequences for the effectiveness of developing countries' national 
literacy education policies. Reasons for the lack of data include the popular 
character of adult literacy programs with their grassroots, bottom-up, 
participatory approach and the fact that, in some countries, the private 
sector has taken over adult literacy. The knowledge base on adult literacy 
programs must be improved and broadened to enable policymakers and donor 
agencies to understand literacy programs' context, grasp learners' needs, and 
discuss programs' effectiveness and relevance by capturing the continuum of 
literacy abilities that goes beyond the simplistic literacy/illiteracy 
dichotomy. Effective monitoring and evaluation are needed to ensure the 
quality of adult literacy programs. The International Adult Literacy Survey 
is a good demonstration of how to use the best of education assessment 
technology to gather data on a broad range of individual literacy skills and 
examine basic skills from a continuum or scale perspective. In assessing 
requests for assistance, donor agencies are interested in research, 
evaluation, and statistics regarding the planning and implementation of adult 
literacy programs. (MN) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



cn 

O 

O 

in 

cn 



p 



w 




hlAMBURG 

1997 



A series of 29 booklets 
documenting workshops 
held at the Fifth 
International Conference 
on Adult Education 



3d Research, evaluation & statistics 



Literacy research, evaluation and statistics 



~s,£ 

o Minor changes have been made to 
improve reproduction quality. 

orriciai OERI position or policy. 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 



I 






TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 



o 

^d. o 

v)ERlC 




2 





This publication has been produced by the UNESCO Institute for Education within 
the context of the follow-up to the Fifth international Conference on Adult Education 
(CONFINTEA V), held in Hamburg in 1997. 

Readers are reminded that the points of view, selection of facts, and the opinions 
expressed in the booklets are those that were raised by panellists, speakers and par- 
ticipants during the workshop sessions and therefore do not necessarily coincide with 
official positions of the UNESCO or of the UNESCO Institute for Education Hamburg. 
The designations employed and the presentation of the material In this publication do 
not imply the expression of any opinion whatsoever on the part of the UNESCO 
Secretariat concerning the legal status of any country or territory, or Its authorities, 
or concerning the delimitations of the frontiers of any country or territory. 



Theme 3: Ensuring universal rights to literacy and basic education 
Booklets under this theme! 

3a Literacy in the world and its major regions 
3b Literacy and learning strategies 
3c Literacy, education and social development 
3d Literacy research, evaluation and statistics 
3e Literacy in multilingual/intercultural settings 
3f Literacy and technology 
3g Literacy for tomorrow 



UNESCO Institute for Education Tel.! (-h49 40) 44 80 41-0 

Feldbrunnenstrasse 58 Fax! (+49 40) 410 77 23 

D-20148 Hamburg 

Germany e-mail! ule@unesco.org 

homepage! http!//www. education. unesco.org/uie 

ISBN 92 82010 89-9 

Design by Matthew Partridge, Hamburg 
Printed by Druckerel Seemann, Hamburg 

1999 




BEST COPY AVAILABLE 

3 

o 

ERIC 



F o ne wo rd 



In July 1997 the Fifth International Conference on Adult Education was 
held in Hamburg, organised by UNESCO and in particular the UNESCO 
Institute for Education, the agency's specialist centre on adult learning 
policy and research. Approximately 1500 delegates attended from all 
regions of the world, with representatives of 140 member states and 
some 400 NGOs. In addition to the work of the commissions and plenary 
which debated the official documents of the Conference The Hamburg 
Declaration and The Agenda for the Future, there were 33 workshops 
organised around the themes and sub-themes of the Conference. 

As part of its CONFINTEA follow-up strategy, the UNESCO Institute 
for Education has produced this series of 29 booklets based on the pre- 
sentations and discussions held during the Conference. The recordings 
of all the workshops were transcribed and synthesized over one year, 
edited, and then formatted and designed. A tremendous amount of work 
has gone into this process. Linda King, coordinator of the monitoring 
and information strategy for CONFINTEA, was responsible for oversee- 
ing the whole process. Madhu Singh, senior research specialist at UIE, 
undertook the mammoth task of writing almost all the booklets based 
on an analysis of the sessions. She was helped in the later stages by 
Gonzalo Retamal, Uta Papen and Linda King. Christopher McIntosh was 
technical editor, Matthew Partridge designed the layout and Janna 
Lowrey was both transcriber and translator. 

The booklets are intended to draw out the central issues and con- 
cerns of each of the CONFINTEA workshops. They are the memory of 
an event that marked an important watershed in the field of adult learn- 
ing. We hope that they will be of use both to those who were able to 
attend CONFINTEA V and those who were not. We look forward to your 
comments, feedback and continuing collaboration with the UNESCO 
Institute for Education. 

Paul Belanger, 

Director, UNESCO Institute for Education, Hamburg 
and Secretary General of CONFINTEA 
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Literacy research, 
evaluation and statistics 



I ntnod ucti o n 

The area of literacy is one domain in the broad field of education where 
relatively little attention has been given to research, evaluation and 
statistics. There is now a critical need to build a more effective know- 
ledge base, to facilitate the design, planning and implementation of 
literacy work. 

This booklet describes some of the issues concerning research, evalu- 
ation and statistics in relation to literacy which were discussed at a 
workshop held during UNESCO's 1997 Fifth International Conference 
on Adult Education (CONFINTEA) in Hamburg. The discussions covered 
a wide geographical and thematic range. The panel was chaired by Dan 
Wagner, Director of the International Literacy Institute, University of 
Philadelphia. The speakers on the panel were! Maki Hayashikawa, 
UNESCO; D. J. Daswanl, Professor of Linguistics, NCERT, India; Sissel 
Volan, Norwegian Development Agency (NORAD); Scott Murray, 
Statistics Canada, Ottawa,* Claudio de Moura Castro, Division of Social 
Policy, Inter-American Development Bank, Washington,’ lla Goksal, 
Director of the Mother Child Education Foundation, Istanbul, Turkey. 

CONFINTEA attracted a very large number of organisations dedicated 
to research and action in adult literacy. It is not enough to say that 
governments do not take or support adult literacy seriously enough. The 
spadework of providing them with research, evaluation and statistics 
upon which they can base their policies is a key task if adult education 
is to reach those It Is most intended for. 
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Sta ting the problem 



Although there are adult literacy programmes in almost every country of 
the developing world, there is a serious lack of comprehensive infor- 
mation on their capacity, performance and Impact. This has direct 
consequences for the effectiveness of national policies. Government and 
official adult education agencies use statistics of adult illiteracy which 
are often not very dependable and which tend to be out of date and too 
general. 

Evaluation is normally one of the formal requirements for running a 
programme. Every programme should provide an evaluation report on 
completion. Very often the commissioning or government agency is 
interested in obtaining value for money. In this respect, literacy 
evaluations have tended to be rather limited in their criteria. In addition, 
each evaluation is based on its own methodology, hypothesis and theory, 
with the result that evaluation reports seldom lend themselves to 
comparisons. 

With regard to research, there are many universities offering courses 
in adult learning, from undergraduate to post-graduate and doctoral 
research programmes. But often there seems to be little interface 
between the programme and the regular university system. University 
graduates seldom become involved in the implementation of adult 
literacy programmes. Consequently, research at the university level is 
largely academic, is treated as a course requirement, and seldom deals 
with field level problems that are faced by implementing agencies. While 
there have been a number of research projects on infrastructure and 
the profiles of learners and teachers, these are sporadic and unconnected 
with each other. 

In view of the inconsistency in the collection, analysis and dissemin- 
ation of Information on literacy, it has been very difficult to demonstrate 
the efficiency of literacy programmes and the quality of the output. Policy 
makers and donors agencies have not been getting the data on which to 
base policies, projects and advocacy. 
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Analysing facts 



One of the reasons for this lack of data lies In the popular character of 
adult literacy programmes with their grassroots, bottom-up participa- 
tory approach. Well analysed facts to support them have not been there. 
For example there has been little comprehensive testing of participant 
achievement or other research that would help to plan and implement 
programmes elsewhere, or academics to deepen theories of effective 
adult learning. The situation is further complicated by the fact that in 
some countries, the private sector has taken over adult literacy, with a 
strong reliance on media. It is too early for any reliable indicators of 
success of these efforts. 

This situation is amply demonstrated by the three large-scale literacy 
movements in Brazil. The first cycle of literacy programmes was driven 
by a peasant social movement. The second cycle came from a military 
regime. The third cycle is being promoted by employers and their 
associations. 

In the first cycle, the literacy drive was connected with wide mobilis- 
ation of peasants. It used a method developed by Paulo Freire to educate 
poor peasants and was part of the process of creating political awareness. 
Literacy was expected to help people become aware of their situation, 
and the idea behind the mobilisation was that, in order to make the 
population literate, it was necessary to make people conscious of their 
situation. 

The second cycle came during the 70s, when Brazil had a growth rate 
of 11 per cent but 30 per cent illiteracy. Economists were hired, and a lot 
of money was spent on creating an extremely well-trained staff of 
teachers and instructors. But this technocratic approach did not work 
because it was too short-sighted, and the emphasis was more on showing 
results rather than producing the right outcome. It failed to consider the 
real difficulties at hand. There was a lack of visibility of the results of 
the many large scale literacy programmes. 

Since the late 80s, Brazilian firms have become leaner and more 
worried about costs and efficiency. This movement has generated a third 
cycle of literacy programmes in the form of a spontaneous development 
inside business and in employers' associations. Although there is no 
master plan, nor a broad consensus, there appears to be a tacit agreement 
on the need for more education and the provision of this by employers. 
This tacit agreement has generated a number of solutions. 
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These include! 



■ Workers returning to school with financial support from their 
employers. 

■ Associations of employers creating programmes like FIEMG 
(The Federation of Industries of Minas Gerais) which has made 
concrete plans to eradicate illiteracy from the State s work force 
by the year 2000. 

■ Payment of 35 million dollars by the Federation of Industries of 
the State of Sao Paulo to a television network to create a televised 
course to prepare for the equivalent of primary- and secondary- 
school leaving examinations as well as offering a course on 
mechanical technology. All lessons take place In factories and 
offices, and involve workers and managers rather than the 
traditional classroom teachers and pupils. 

Need for a better knowledge base 

We need a much wider knowledge base and better research data upon 

which to base adult literacy programmes. The questions that need to be 

addressed are! 

■ Who participates in literacy programmes? 

■ How do adults learn? 

■ What knowledge do they bring with them when they 
come to the programme? 

■ What Is their motivation? 

■ What is the communicative repertoire of the learner? 

■ What is their mother tongue? 

■ What is their second language? 

■ What are the uses of literacy in society? 

■ How is literacy retained? 

■ How much literacy is retained ? 

■ How is literacy diffused? 

■ What are the changes In literacy requirement? 

■ What is the Impact of literacy on productivity, 
quality of life, and on society? 
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The purpose is to understand the context, grasp the learners' felt 
needs and to discuss the effectiveness and relevance of programmes by 
capturing the continuum of literacy abilities that goes beyond the 
simplistic literacy/illiteracy dichotomy. 



Ensuring quality of adult literacy 
programmes through effective 
monitoring and evaluation 

Providers of information on literacy can be individual persons and 
literacy centres/institutions. Data on individual persons can be obtained 
from population censuses, household surveys, and literacy surveys. The 
type of data collected through population and household surveys would 
include data on the illiterate and literate population, educational 
attainment, school attendance, household and personal characteristics. 
Household and literacy follow-up surveys could include more detailed 
questions on perceived relevance and quality of literacy programmes 
attended, retention, use, spread, improvement of acquired skills and 
finally outcomes and impacts of literacy. 

Data channels for monitoring literacy institutions and programmes 
are regular surveys or administrative records and reports. Such data 
focus on the availability, location and timing of adult literacy programmes 
by type of institution, capacity, participation, level of resource inputs, 
quality of personnel and facilitators, relevance and impact of the pro- 
gramme and level of achievement of participants through participatory 
learner assessment. 

Literacy Information management systems, including practical 
assessment methodology, will provide information on three levels of 
programmes and a continuum of literacy abilities. These programmes 
are! 

■ Non-formal basic education for out-of school children and youth. 

■ Adult basic literacy programmes. 

■ Continuing education. 
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In terms of the subsequent analysis of the information, it will be par- 
ticularly valuable that the following issues be clarified! 

■ learning achievement in individuals! 

■ outcomes/impacts on learners' everyday lives! 

■ programme level efficiency and accountability! 

■ in-country and international comparisons. 

T he coming years will witness the development of broader strategies for 
monitoring literacy, by! 

■ making coordinated use of different approaches! 

■ introducing a participatory approach! 

■ promoting the creation of national directories/databases! 

■ fostering a culture of information for literacy! 

■ developing practical literacy assessment methodologies and tools. 

To do so, the countries may have to adopt the following measures and 
policies! 

■ building in-country monitoring capacities! 

■ maximum use of existing monitoring channels! 

■ establishing universal literacy criteria and norms! 

■ adapting the assessment methodology to specific social and 
cultural contexts! 

■ applying the methodology to specific countries! 

■ mobilising appropriate partners, such as NGOs and 
community organisations. 
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Surveys on literacy skills 



The International Adult Literacy Survey (lALS), based on an initial 
survey in Canada and a subsequent survey in the United States, and 
which has now been carried out in many countries, is a good demon- 
stration of how to use the best of education assessment technology to 
gather data on a broad range of individual literacy skills and to look at 
basic skills from a continuum or scale perspective, rather than reducing 
literacy research simply to categories such as literacy/illiteracy. The 
survey has shown that! 

■ Factors which make reading more or less difficult for adults 
are common to many languages and cultures. 

■ The conditions and causes of literacy skills are becoming clearer. 

■ Such studies offer a normative base, against which all other 
statistics can be compared. 

■ They give a coherent framework for thinking about literacy 
development. 

■ They inform choices at both the individual and the governmental 
policy level. Individuals need to make choices about what they 
want to do with their lives) governments need to make choices 
about resource allocations. 

There are several myths about literacy which the literacy survey has 
tried to demystify. It is often assumed, for example, that literacy is very 
high in the industrialised countries. The literacy survey has been able 
to show that in every country studied in the lALS survey there is a high 
percentage of adults without the required skills for effectively partici- 
pating in the society and economy. Low levels of skills limit the social 
and economic participation and opportunity of adults. 

There is the myth that literacy is a product of formal schooling and 
that initial education gives everybody a standard quantum of literacy 
competency. Data is now available indicating that this is not true. There 
are many people who are illiterate despite their formal qualifications. 
On the other hand, there are a great number of adults who do not have 
formal qualifications, yet have literacy skills at quite high levels. This 
indicates that there are other forces in society besides formal education 
that are responsible for a persons level of literacy. It also raises questions 
about equity and quality in the educational system and the role of 
literacy environments, workplace literacy and family literacy. 
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Information from highly-developed economies shows that literacy 
skills are to some extent independent of formal qualification! only 30 
per cent of literacy can be attributed to formal education and qualifi- 
cations, while a further 30 per cent is acquired In out-of-school situ- 
ations. The same is true of the workers who are past the credential stage 
in our economies. 

Literacy has a strong Influence on who gets access to adult continuing 
learning, particularly at the workplace. Literacy skills can thus be a key 
marginalising force, leaving governments with the problem of what to 
do with those people who are marginalised In the process. 

Literacy is not just a matter of the supply of skills, but also of the 
demand for skills in the economy. This is Important to recognise because 
it is often held that the workforce is over qualified and does not use the 
skills that it already has. 

Many economies pride themselves on having an equitable system - 
educationally and economically - with fairly reasonable equality of 
opportunity. But, contrary to this view, analysis from the surveys shows 
that children and adults from disadvantaged backgrounds who live in 
provinces with severe social and economic problems perform more 
poorly on literacy achievement than adults and children from other 
provinces. 

The major problem in the world today is not one of absolute illiteracy 
but of low literacy skills which are insufficient to meet the demands of 
a changing economy. 
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The importance of research on adult 
learning and literacy 



A research project conducted in Turkey has been developed into a full- 
fledged programme of parent and early childhood education, called the 
Mother-Child-Education programme. It is a multiple-pronged programme, 
which not only underlines the potential of academic research, but also 
attempts to create a more literate family environment, for both parent 
and child. It sets out to assess the socio-economic, socio-emotional and 
cognitive development of pre-school children and the impact of training 
given to a random sample of mothers for over two years. There were two 
components to the mother training project! (1 ) A cognitive component, 
which was designed to foster cognitive development of the child; (2) A 
mother support component, which was designed to support the mother 
in fostering the socio-emotional and personality development of the 
child. 

The cognitive component was designed to create a more literate home 
environment for mother and child. It was found that the implementation 
of the cognitive programme for children contributed to significant differ- 
ences in children s cognitive development and in mothers* literacy skills. 

This programme, implemented in 50 provinces in Turkey, has been 
extended to include reproductive and family health, aiming not only to 
upgrade the pre-literacy and numeracy skills of a child, but also aiming 
to further the literacy of the mother in order to support a more func- 
tionally literate society. 

Literacy needs to be supported by other institutions in society, the 
family being one of the most important. People need to be reached in 
the context of their environment and positions in society. Mothers are 
being reached and drawn into the programme by telling them! ’’You can 
help prepare your children for school, or you can help them to become 
more successful in school." This is found to be a better approach than 
saying! ’’Your skills are not enough for the society - let us try and upgrade 
them.** 

The three main aspects highlighted by the study on mother-child 
literacy are! (1) The importance of research which guided the programme 
and enabled it to reach a wide range of people; (2) The importance of 
multi-purpose programmes in literacy, with an emphasis on adult 
education rather than on adult literacy alone; (3) The importance of 
conducting evaluation at different levels and for different stakeholders. 
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Donor agencies and literacy programmes 



In assessing requests for assistance, donor agencies are interested in 
research, evaluation and statistics in the planning and implementation 
of adult literacy programmes. In planning such programmes the following 
considerations are important; 

■ Planning needs to be accompanied by systematic research or 
solid evaluation. 

■ A closer contact needs to be made between academic research 
in universities and literacy programmes. 

■ Official agencies should provide reliable statistics on literacy, 
emphasising both quantitative and qualitative data. 

■ It is necessary to take into account the growing role of industry 
In providing literacy education. 

■ Improved tools are needed to enhance the knowledge base on 
literacy. It is not literacy or formal qualifications per se that 
make people literate. It is more a question of providing equitable 
opportunities for continuous learning. 

■ Learners need a supportive literate environment for sustainability 
of literacy skills and for developing themselves into holistic 
personalities. 

Effective action depends on adequate knowledge. The very sustain- 
ability of literacy work in an Increasingly competitive educational 
environment will require adult literacy and adult learning to build a 
better knowledge base on initial education. 
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This document can be freely reproduced. It would be appreciated if 
a copy of any publication reproducing this text in full or in part could 
be sent to! Publications Department, UNESCO Institute for Education. 




The CONFINTEA logo, designed by Michael Smitheram 
of Australia, represents the lines on the palm of a hand. 
These lines are universal and yet different for each 
sukyect. They celebrate cultural diversity and thejoy 
of learning. 
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